I have a large text file (5GB) and three key files. My job is to counts each key in text file.
Due to larger file size, I split this text file into multiple small chunks. The problem is first few letters of a key exists in one part and rest other letters of key is in another part. For example “Streaming Issue” is a key, here “str” exist in part1 & “eaming Issue” in part2. Each small chunks will be read by a separate thread.
Then how to counts occurrences of entire key?