Question

Data in user_migrations table is capped at 250K in S3 Connection

  • 20 May 2022
  • 1 reply
  • 29 views

Badge

Hey everyone.

We are using S3 Connection to load data to our Lakehouse. We implemented the identity resolution as per documentation here https://help.heap.io/heap-connect/heap-connect-guide/identity-resolution/

What we have noticed is that the user_migrations table always contains only up to 250K rows and as the table strategy for ingestion is not incremental, we are missing on some of the identity resolution for our events and sessions.

Is this expected behaviour and known limitation? We haven’t found anything in the documentations mentioning this. Is there a workaround we could rely on?

Thanks.


1 reply

Userlevel 5
Badge +3

Hey @vojtakopal — In our S3 connector, we chunk our data into multiple files. For user_migrations these files happen to be 250K rows apiece. It may be that your job is not checking for and ingesting all available user_migration files.

 

Source: https://help.heap.io/integrations/data-warehouses/s3/

 

If you don’t have multiple files or there is some other issue, please reach out to support@heap.io. 

Reply