Skip to content

Martini Cursors

Overview

Martini Cursors are memory-efficient pointers that let you process millions of data records without loading everything into memory simultaneously. Instead of storing all data at once, cursors replace old records with new ones as you iterate through large datasets, preventing memory overflow and maintaining optimal performance.

What You Will Learn

  • What cursors are and how they differ from regular arrays in Martini
  • Where cursors are used throughout the Martini platform
  • How cursors are managed automatically or manually

When To Use This

Use cursors when you need to process large datasets that would otherwise consume excessive memory. Martini automatically creates cursors in specific scenarios, but understanding their behavior helps you design more efficient data processing workflows.

Prerequisites

Understanding Martini Cursors

Cursors appear as arrays in Martini, showing the array overlay icon , but behave differently to optimize memory usage when handling large datasets.

How Cursors Work

Cursors optimize memory usage through specific behavioral constraints and automatic memory management. Unlike regular arrays, cursors have limitations that enable efficient processing of large datasets:

Behavioral Constraints:

  • Single iteration: They can only be iterated over once
  • Read-only cursors: Some cursors can only be read from
  • Write-only cursors: Some cursors can only be written to

Memory Management:

When you process data through a cursor, Martini manages memory automatically by:

  1. Loading only the current record into memory
  2. Processing that record through your workflow or service
  3. Discarding the processed record from memory
  4. Loading the next record to continue processing

Where Cursors Are Used in Martini

When working with large datasets in Martini, you'll encounter cursors in specific scenarios where memory optimization is essential.

Database Operations

When working with database operations that handle large result sets, you'll see cursors being used to manage memory efficiently:

You'll see write-only cursors when:

You'll see read-only cursors when:

File Processing Operations

When processing large files, you'll encounter cursors that stream data without loading entire files into memory:

You'll see cursors with flat file operations:

  • Using flat file reading and writing services for large CSV or delimited files
  • Calling getTextFileInputCursor(...) in File Functions for large text files

You'll see cursors with structured data files:

  • Using openJsonFileInputCursor(...) or openJsonStreamInputCursor(...) in JSON Functions for reading large JSON files
  • Using openJsonOutputCursor(...) in JSON Functions for writing large JSON datasets
  • Using openXmlFileInputCursor(...) or openXmlStreamInputCursor(...) in XML Functions for reading large XML files
  • Using openXmlOutputCursor(...) in XML Functions for writing large XML datasets
  • Using openYamlFileInputCursor(...) or openYamlStreamInputCursor(...) in YAML Functions for reading large YAML files
  • Using openYamlOutputCursor(...) in YAML Functions for writing large YAML datasets

Cursor Management in Martini

Martini provides both automatic and manual cursor management depending on your workflow requirements.

Automatic Cursor Management

When cursors are used with iteration components, Martini handles reading and writing automatically:

These components automatically read from input cursors and write to output cursors without requiring manual cursor operations.

Manual Cursor Management

For advanced scenarios requiring direct cursor control, use cursor-specific functions in Service Functions:

  • Reading functions: Retrieve records from input cursors
  • Writing functions: Add records to output cursors
  • Closing functions: Properly release cursor resources

Manual management gives you precise control over cursor operations when automatic handling doesn't meet your specific requirements.

Cursor Use Cases

Cursors are essential for processing large datasets without overwhelming system memory in these common scenarios:

Processing Millions of Database Records: When querying large tables with millions of customer records, transaction logs, or historical data, cursors enable you to iterate through results without loading the entire dataset into memory. Multi-select SQL operations automatically return cursors to handle these massive result sets efficiently.

Batch Database Operations: When inserting, updating, or processing large batches of data records, cursors allow you to write data incrementally without memory constraints. Batch SQL operations use write-only cursors to stream data directly to the database.

Large Flat File Processing: When reading or writing large CSV, TSV, or other delimited files containing millions of rows, cursors stream data line by line. This enables processing files larger than available system memory without performance degradation.

Structured File Streaming: When working with large JSON, XML, or YAML files containing extensive data arrays or complex nested structures, cursors parse and process data incrementally. This prevents memory overflow when handling files with thousands of objects or deeply nested hierarchies.

Log File Analysis: When processing large application logs, system logs, or audit trails stored in various formats, cursors enable continuous processing of log entries without loading entire files into memory, maintaining system performance during analysis operations.

Helpful Resources