Read Large Files in .NET

If the file we want to process from a network location is huge and doesn't fit in the memory, we can read just the header of the file and read the body as stream. We can either pass the HttpCompletionOption to the GetAsync method or use GetStreamAync method:

var response = await httpClient.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);
var stream = await response.Content.ReadAsStreamAsync();
using var sr = new StreamReader(stream);

OR

var stream = await httpClient.GetStreamAsync(url);
using var sr = new StreamReader(stream);

string? line;
while((line = sr.ReadLine()) != null)
{
	Console.WriteLine(line);
}

In this approach even if the file is 100GB, we're not loading the entire file into memory and we'll only read the file in small chunks. You can confirm this by looking at your application memory by GC.GetTotalMemory(). If you don't pass ResponseHeaderRead parameter in GetAsync method, you'll notice that your total memory will be increased by the size of the file.