Too much data in Contains (Linq): How to increase performace

I have a linq query like this:

from a in context.table_A
   join b in
   (from temp in context.table_B
   where idlist.Contains(temp.id)
   select temp)
   on a.seq_id equals b.seq_id into c
   where
   idlist.Contains(a.id)
   select new MyObject
   {
     ...
   }).ToList();

idlist is List
The problem I have is that the idlist has too many values (hundreds of thousands to several million records). It works fine with few records but when there are too many records, the contains function is error.

Error log is

The query plan cannot be created due to lack of internal resources of
the query processor. This is a rare event and only occurs for very
complex queries or queries that reference a very large number of
tables or partitions. Make the query easy.

I want to improve the performance of this section. Any ideas?

Contents hide

Answers:

Method 1

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

I would suggest to install extension linq2db.EntityFrameworkCore and use temporary tables with fast BulkCopy

public class IdItem
{
   public int Id { get; set; }
}

...

var items = idList.Select(id => new IdItem { Id = id });

using var db = context.CreateLinqToDBConnection();
using var temp = db.CreateTempTable("#IdList", items);

var query =
   from a in context.table_A
   join id1 in temp on a.Id equals id1.Id
   join b in context.table_B on a.seq_id equals b.seq_id
   join id2 in temp on b.Id equals id2.Id
   select new MyObject
   {
      ...
   };

// switch to alternative translator
var query = query.ToLinqToDB();

var result = query.ToList();

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0

0 0 votes

Article Rating