How many times each word has occurred in the text file

Assignment Help Python Programming
Reference no: EM131215031

You have all experienced how when you are typing a text message the application will provide potential words that complete what you are typing (and sometimes insist on completing them incorrectly). You will write a Python program that is capable of generating a likely set of completion words given the start of a word as input to the program.

In order to be able to make these suggestions, your program will first need to analyze a large amount of text (like a book) to get an idea of what words occur how often in regular text. Once this analysis is made, then it will be possible to do a quick lookup into a data structure described below to generate the potential words.

Step 1 - Text file analysis

Ask the user for the name of a file that contains the text to analyze. Open the file and read the lines of text. Break each line into words, stripping out all of the special symbols. For each word that you find that is longer than one character, do the following. First, add the word to a dictionary of words you have seen so far. The values in this dictionary should be a count of how many times each word has occurred in the text file. Second, add the word into a data structure that is keeping track of word information for the completion process. This data structure has 3 basic parts. The top level part is a list that will have as many elements in it as the number of characters in the longest word seen so far. Each element in this list will be a dictionary which is indexed by the 26 letters of the alphabet. The value for each entry in the dictionary will be a set of words. A word is added into this structure as follows. The word is first added to the set contained in the dictionary corresponding to slot 0 in the list, indexed by the key of the first letter in the word. Then, the second letter of the word is used to index into the dictionary corresponding to slot 1 in the list to add the word to that set. This continues for the remaining letters in the word. So this means that the word "help" would be added to the set of words that have an h as their first letter, the set having e as the second letter, the set for l as the third letter, and then p as the fourth letter.

Step 2 - Word completion

With these two structures of information constructed based on the input file, it is time to do word completion. Within a loop that can continue as long as the user wants, ask to read the starting portion of a word. You then must determine the set of all possible words you have seen that begin with this string of characters. This set of words can be computed by intersecting a group of sets extracted from the data structure. These sets are found by indexing into the dictionary corresponding to the character value of each position in the string. You can intersect all of these sets together two at a time to get your result. So if the string "he" was typed in, you would intersect the set of words that have h as their first letter with the set of words that have e as their second letter. This gives the set of words that start with "he".

Next you must rank the possible set of words based on how often they were found in the text, assuming that words that occur often are more likely to occur again. For each word in the final set of possibilities, extract the occurrence count from the other dictionary you constructed. Then sort the words based on these values. Print out the top five candidate words (or fewer if less than five exist) with a percentage of how likely each word is the proper completion of the string given. To compute percentages, first total up all of the occurrences of the words in the set, then divide the count for each by this amount.

Submission

Submit your commented source code. Acquire a few large text files to test your program. Plain text files of books are a good suggestion, like the War & Peace file we used in class. Things like Project Gutenberg are a good source for these. Submit some sample output from your program using one of these files.

Reference no: EM131215031

Questions Cloud

Challenges of securing information : Create a 7-10-slide PowerPoint Presentation on the challenges of securing information, and list some of today's information security attacks. You may use various sources (scholarly). Be sure to cite any sources used in a reference slide with prope..
Determine the current failure intensity : Assume that a program will experience 200 failures in infinite (6) time. It has now experienced 100. The initial failure intensity was 20 failures/CPU hr. (i) Determine the current failure intensity
What value for constants a and b lead to new random variable : Prove that Y is also Gaussian for any a ≠ 0 . - What values for the constants a and b will lead to the new random variable Y having zero mean and unit variance?
Recognise the importance of bloom taxonomy : By the end of this topic you should be able to.. Describe difference between educational aims & learning  outcomes and Explain the importance of learning outcomes
How many times each word has occurred in the text file : The values in this dictionary should be a count of how many times each word has occurred in the text file. Second, add the word into a data structure that is keeping track of word information for the completion process.
Swapping the processes : What are conditions on which deadlock can occur while swapping the processes?
Accomplish my goals sincerely : 1. Overview of the companies you learned about 2. Profile of your selections and the reasons you chose them 3. Summary of why you believe this will help me accomplish my goals Sincerely,Harrison Kirby
Estimate the average number of students in a course : For each university, estimate the average number of students in a course.
Data of the current instruction : What holds the data of the current instruction then copies it to the IR. I think its connected to data bus's?

Reviews

Write a Review

 

Python Programming Questions & Answers

  Python function to calculate two roots

Write a Python function main() to calculate two roots. You must input a,b and c from keyboard, and then print two roots. Suppose the discriminant D= b2-4ac is positive.

  Design program that asks user to enter amount in python

IN Python Design a program that asks the user to enter the amount that he or she has budget in a month. A loop should then prompt the user to enter his or her expenses for the month.

  Disabled individuals open jars and containers

What variables are you going to need and what will be their datatype and how would you break this problem into numerous smaller units?

  Turn the turtle image into a .gif picture

How can you turn the turtle image into a .gif picture when using the built-in turtle for Python? If that's impossible how do you remove the line when you move the turtle around?

  Create a ride share simulator assignment

Create a ride share simulator assignment. In this particular function, I am trying to create a list of the drivers in a Driver Class that are currently listed as idle. Since they are idle, I can then match them up with an awaiting rider. The assignme..

  Create a python program that allows user enter class name

Create a Python program that allows user enter class name, faculty name, name of each student, and letter grade each student earned.

  Create mini-python projects or programs

Your assignment is to create mini-Python projects or programs and to run them in the Python-IDLE environment

  A string is valid windows filename.

writing a function in python that verifies whether a string is valid windows filename.

  Implement core security features

What you will be implementing here is inspired by real world cases such as the Conficker work. To have any chance of defeating such a threat, you need to understand how they defend themselves.

  Python errors

python errors, please correct them that are located in this program,

  Write python program isosceles tri equilateral tri rectangle

Write python program Isosceles Tri Equilateral Tri Rectangle, Write another "driver" script called project1.py which imports the polygon.py module, reads an input file of polygonal data and writes another file of areas and perimeters.

  Write a function trans(m) which returns the transpose

Write a function trans(M) which returns the transpose of an n-by-n matrix M. The matrix M is represented by a list of n lists, each of length n. Transposing M means that each M[i][j] is swapped (once!) with M[j][i].

Free Assignment Quote

Assured A++ Grade

Get guaranteed satisfaction & time on delivery in every assignment order you paid with us! We ensure premium quality solution document along with free turntin report!

All rights reserved! Copyrights ©2019-2020 ExpertsMind IT Educational Pvt Ltd