tokenizer

tokenizer
A system that parses an input stream into tokens.

Wikipedia foundation.

Игры ⚽ Нужен реферат?

Look at other dictionaries:

  • Tokenizer — Ein Tokenizer (auch: lexikalischer Scanner, kurz Lexer) ist ein Computerprogramm zur Zerlegung von Plain text (zum Beispiel Quellcode) in Folgen von logisch zusammengehörigen Einheiten, so genannte Token (engl. tokens). Als solcher ist er oft… …   Deutsch Wikipedia

  • Atari BASIC — For the version of BASIC bundled with the Atari ST computer series, see Atari ST BASIC . ATARI BASIC is a ROM resident BASIC interpreter for the Atari 8 bit family of 6502 based home computers. The interpreter originally shipped on an 8 KB… …   Wikipedia

  • Lexical analysis — In computer science, lexical analysis is the process of converting a sequence of characters into a sequence of tokens. Programs performing lexical analysis are called lexical analyzers or lexers. A lexer is often organized as separate scanner and …   Wikipedia

  • PHPDoc — is an adaptation of Javadoc for the PHP programming language. It is a formal standard for commenting PHP code. It offers three main advantages to generic or random commenting styles: First, it makes comments readable in a standard method… …   Wikipedia

  • Comparison of layout engines (HTML5) — HTML HTML and HTML5 Dynamic HTML XHTML XHTML Mobile Profile and C HTML Canvas element Character encodings Document Object Model Font family HTML editor HTML element HTML Frames HTML5 video HTML scrip …   Wikipedia

  • Commodore BASIC — Commodore BASIC, also known as PET BASIC, is the dialect of the BASIC programming language used in Commodore International s 8 bit home computer line, stretching from the PET of 1977 to the C128 of 1985. The core was based on 6502 Microsoft BASIC …   Wikipedia

  • ThinBasic — Infobox programming language name = ThinBasic developer = Eros Olmi and Roberto Bianchi latest release version = v1.6.0.10 latest release date = July 5 2008 dialects = BASIC operating system = Windows license = Freeware website =… …   Wikipedia

  • Index (search engine) — Search engine indexing collects, parses, and stores data to facilitate fast and accurate information retrieval. Index design incorporates interdisciplinary concepts from linguistics, cognitive psychology, mathematics, informatics, physics, and… …   Wikipedia

  • General Architecture for Text Engineering — Infobox Software name = GATE caption = General Architecture for Text Engineering. developer = [http://gate.ac.uk/ GATE research team] , Dept. Computer Science, University of Sheffield released = 1996 frequently updated = yes programming language …   Wikipedia

  • CMS Made Simple — Not to be confused with CMSimple. CMS Made Simple CMS Made Simple Admin Portal …   Wikipedia

Share the article and excerpts

Direct link
Do a right-click on the link above
and select “Copy Link”