tokenize

tokenize
To reduce to a set of tokens by parsing.

Wikipedia foundation.

Игры ⚽ Нужно решить контрольную?

Look at other dictionaries:

  • Лексический анализ — Запрос «Лексема (информатика)» перенаправляется сюда; см. также другие значения. В информатике лексический анализ  процесс аналитического разбора входной последовательности символов (например, такой как исходный код на одном из языков… …   Википедия

  • Лексер — В информатике лексический анализ процесс аналитического разбора входной последовательности символов (например, такой как исходный код на одном из языков программирования) с целью получения на выходе последовательности символов, называемых… …   Википедия

  • Лексический анализатор — В информатике лексический анализ процесс аналитического разбора входной последовательности символов (например, такой как исходный код на одном из языков программирования) с целью получения на выходе последовательности символов, называемых… …   Википедия

  • Applesoft BASIC — was a dialect of BASIC supplied on the Apple II computer, superseding Integer BASIC. Applesoft BASIC was supplied by Microsoft and its name is derived from the names of both Apple and Microsoft. The first version of Applesoft was released in 1977 …   Wikipedia

  • GFA BASIC — is a dialect of the BASIC programming language, by Frank Ostrowski. The first version was finished in 1986. In the mid and late 80 s, it became very popular for the Atari ST homecomputer range (since the ST BASIC shipped with them was more… …   Wikipedia

  • TI-BASIC — is the unofficial name of a BASIC like language built into Texas Instruments (TI) s graphing calculators. , , (Except for the TI 85 and TI 86 variants, TI only refers to it as the calculators extensive programming capability .) Though the TI… …   Wikipedia

  • Tag (metadata) — For other uses, see Tag. A tag cloud with terms related to Web 2.0 In online computer systems terminology, a tag is a non hierarchical keyword or term assigned to a piece of information (such as an Internet bookmark, digital image, or computer… …   Wikipedia

  • Just BASIC — is a dialect of the highly popular BASIC programming language of the 1970 s for 32 bit Windows computer systems. It s the freeware version of Liberty BASIC, popular since 1992. It allows you to create programs with a comprehensive tutorial, help… …   Wikipedia

  • Repetition blindness — (RB) is a phenomenon observed in Rapid Serial Visual Presentation. Subjects are less likely to detect the repetition of a target stimulus than they are to detect a second, different target.For example, a subject s chances of correctly reporting… …   Wikipedia

  • Shunting yard algorithm — The shunting yard algorithm is a method for parsing mathematical equations specified in infix notation. It can be used to produce output in Reverse Polish notation (RPN) or as an abstract syntax tree (AST). The algorithm was invented by Edsger… …   Wikipedia

Share the article and excerpts

Direct link
Do a right-click on the link above
and select “Copy Link”