Handling large files with millions of rows in Node.js can be a challenging yet essential task, especially in scenarios like data migration, analytics processing, or batch uploads. In this blog post, we’ll explore strategies and code snippets for efficiently uploading such massive files using Node.js, focusing on data chunking and queues for optimal performance.

Understanding the Challenge

Uploading large files can strain server resources and impact application responsiveness. Common challenges include memory usage spikes, timeouts, and potential crashes. However, breaking down the file into manageable chunks and utilizing queues can mitigate these issues effectively.

Using Data Chunking

Data chunking involves dividing a large file into smaller, more manageable parts, processing them sequentially or concurrently, and then aggregating the results. Here’s a simplified example using Node.js:

const fs = require('fs');
const readline = require('readline');

async function processLargeFile(filePath) {
    const fileStream = fs.createReadStream(filePath);
    const rl = readline.createInterface({
        input: fileStream,
        crlfDelay: Infinity,
    });

    let chunk = [];
    for await (const line of rl) {
        chunk.push(line);
        if (chunk.length === 1000) {
            await processChunk(chunk);
            chunk = [];
        }
    }

    // Process any remaining lines
    if (chunk.length > 0) {
        await processChunk(chunk);
    }
}

async function processChunk(chunk) {
    // Your processing logic here
    console.log(`Processing ${chunk.length} lines`);
    // Example: Insert into a database or perform data manipulation
}

// Usage
const filePath = 'path/to/your/largefile.csv';
processLargeFile(filePath);

In this example, we’re reading a large file line by line and processing chunks of 1000 lines at a time. Adjust the chunk size based on your system’s memory and processing capabilities.

Leveraging Queues

Queues help manage tasks asynchronously, allowing for better resource utilization and preventing overload. We’ll integrate a simple queue using async/await and Promise in Node.js:

class TaskQueue {
    constructor(concurrency) {
        this.concurrency = concurrency;
        this.running = 0;
        this.queue = [];
    }

    pushTask(task) {
        this.queue.push(task);
        this.next();
    }

    async next() {
        while (this.running < this.concurrency && this.queue.length) {
            const task = this.queue.shift();
            this.running++;
            try {
                await task();
            } catch (error) {
                console.error(error);
            }
            this.running--;
        }
    }
}

// Usage
const queue = new TaskQueue(2); // Adjust concurrency based on system limits

function processChunk(chunk) {
    return new Promise((resolve, reject) => {
        // Your processing logic here
        console.log(`Processing ${chunk.length} lines`);
        // Simulating async task completion
        setTimeout(() => {
            resolve();
        }, 1000);
    });
}

// Simulating chunks
const chunks = [[1, 2, 3], [4, 5, 6], [7, 8, 9]];

chunks.forEach(chunk => {
    queue.pushTask(() => processChunk(chunk));
});

In this queue implementation, tasks are executed concurrently up to the specified concurrency level. Adjust concurrency based on your server’s capacity and workload.

Uploading very large files with millions of rows in Node.js requires careful handling to avoid performance issues. By employing data chunking and leveraging queues, you can effectively manage memory usage, optimize processing speed, and ensure a smooth upload experience for users. Tailor these strategies to your specific use case and scalability requirements for optimal results.