Solved

Data in user_migrations table is capped at 250K in S3 Connection

  • 20 May 2022
  • 1 reply
  • 40 views
vojtakopal
Badge

Hey everyone.

We are using S3 Connection to load data to our Lakehouse. We implemented the identity resolution as per documentation here https://help.heap.io/heap-connect/heap-connect-guide/identity-resolution/

What we have noticed is that the user_migrations table always contains only up to 250K rows and as the table strategy for ingestion is not incremental, we are missing on some of the identity resolution for our events and sessions.

Is this expected behaviour and known limitation? We haven’t found anything in the documentations mentioning this. Is there a workaround we could rely on?

Thanks.

icon

Best answer by jonathan 27 May 2022, 17:43

View original

1 reply

jonathan
Rank
Userlevel 5
Badge +3

Hey @vojtakopal — In our S3 connector, we chunk our data into multiple files. For user_migrations these files happen to be 250K rows apiece. It may be that your job is not checking for and ingesting all available user_migration files.

 

Source: https://help.heap.io/integrations/data-warehouses/s3/

 

If you don’t have multiple files or there is some other issue, please reach out to support@heap.io. 

Reply


Terms & ConditionsCookie settings