Using any large JSON as a lazy collection in Laravel
If you’re working with Laravel (or any PHP application for that matter), you might have stumbled upon a situation where you need to process a large JSON input. For instance, you might want to use a JSON file to seed your database.
The problem is, that if the JSON is large, it will take a lot of memory to load it into the memory. And eventually, it will result in a memory exhaustion error.
A new package called lazy-json attempts to fix this issue. It allows you to use any large JSON file as a lazy collection in your application. It’s a framework-agnostic package that can be used with any PHP application.
Under the hood, it uses Laravel’s Lazy Collections to achieve this. So, you can use all the methods available in the LazyCollection class.
Installation
You can install the package via composer like so.
composer require cerbero/lazy-json
Usage
There are three ways you can use this package in your project.
use Cerbero\LazyJson\LazyJson;
use Illuminate\Support\LazyCollection;
use function Cerbero\LazyJson\lazyJson;
// auto-registered lazy collection macro
$lazyCollection = LazyCollection::fromJson($source);
// static method
$lazyCollection = LazyJson::from($source);
// namespaced helper
$lazyCollection = lazyJson($source);
The source here can be a string, an array, a file path, a resource, or a URL.
Here’s the list of all the supported sources.
- strings, e.g.
{"foo":"bar"} - iterables, i.e. arrays or instances of
Traversable - file paths, e.g.
/path/to/large.json - resources, e.g. streams
- API endpoint URLs, e.g.
https://endpoint.jsonor any instance ofPsr\Http\Message\UriInterface - PSR-7 requests, i.e. any instance of
Psr\Http\Message\RequestInterface - PSR-7 messages, i.e. any instance of
Psr\Http\Message\MessageInterface - PSR-7 streams, i.e. any instance of
Psr\Http\Message\StreamInterface - Laravel HTTP client requests, i.e. any instance of
Illuminate\Http\Client\Request - Laravel HTTP client responses, i.e. any instance of
Illuminate\Http\Client\Response - user-defined sources, i.e. any instance of
Cerbero\JsonParser\Sources\Source
An example
Here’s a basic example of how to turn JSON into a lazy collection using the global helper function.
use function Cerbero\LazyJson\lazyJson;
$source = '{
"users": [
{
"id": 3,
"name": "Amit Merchant",
"email": "[email protected]"
},
{
"id": 1,
"name": "John Doe",
"email": "[email protected]"
},
{
"id": 2,
"name": "Jane Doe",
"email": "[email protected]"
}
]
}';
$lazyCollection = lazyJson($source, ['users.*.name']);
$mapped = $lazyCollection->map(function (string $value, string $key) {
return $value;
});
$mapped->each(function ($value, $key) {
dump($key . ' => ' . $value);
// iteration 1: "name => Amit Merchant"
// iteration 2: "name => John Doe"
// iteration 3: "name => Jane Doe"
});
As you can tell, the lazyJson helper function takes two arguments.
The first one is the source of the JSON and the second one is the path to the data you want to extract from the JSON. You can pass it as dot-notation syntax to extract the sub-data from the JSON.
If you want multiple paths, you can pass them as an array like so.
$lazyCollection = lazyJson($source, [
'users.*.name',
'users.*.email'
]);
In closing
So, this is how you can use any large JSON as a lazy collection in your Laravel application. I think it’s a pretty handy package to have in your toolkit if you’re working with large JSON files.
Learn more about the package: lazy-json
Like this article?
Buy me a coffee
👋 Hi there! I'm Amit. I write articles about all things web development. You can become a sponsor on my blog to help me continue my writing journey and get your brand in front of thousands of eyes.
