Trending December 2023 # Language Translation With Transformer In Python! # Suggested January 2024 # Top 16 Popular

You are reading the article Language Translation With Transformer In Python! updated in December 2023 on the website Bellydancehcm.com. We hope that the information we have shared is helpful to you. If you find the content interesting and meaningful, please share it with your friends and continue to follow and support us for the latest updates. Suggested January 2024 Language Translation With Transformer In Python!

This article was published as a part of the Data Science Blogathon

Introduction

Natural Language Processing (NLP) is a field at the convergence of artificial intelligence, and linguistics. The aim is to make the computers understand real-world language or natural language so that they can perform tasks like Question Answering, Language Translation, and many more.

NLP has lots of applications in different fields.

1. NLP enables the recognition and prediction of diseases based on electronic health records.

2. It is used to obtain customer reviews.

3. To help to identify fake news.

4. Chatbots.

5. Social Media monitoring etc.

What is the Transformer?

The Transformer model architecture was introduced by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin in their paper “Attention Is All You Need”. [1]

The Transformer model extracts the features for each word using a self-attention mechanism to know the importance of each word in the sentence. No other recurrent units are used to extract this feature, they are just activations and weighted sums, so they can be very efficient and parallelizable.

Source: ” Attention Is All You Need” paper

In the above figure, there is an encoder model on the left side and the decoder on the right. Both encoder and decoder contain a core block of attention and a feed-forward network repeated N number of times.

It has a stack of 6 Encoder and 6 Decoder, the Encoder contains two layers(sub-layers), that is a multi-head self-attention layer, and a fully connected feed-forward network. The Decoder contains three layers(sub-layers), a multi-head self-attention layer, another multi-head self-attention layer to perform self-attention over encoder outputs, and a fully connected feed-forward network. Each sub-layer in Decoder and Encoder has a Residual connection with layer normalization.

Let’s Start Building Language Translation Model

Here we will be using the Multi30k dataset. Don’t worry the dataset will be downloaded with a piece of code.

First the Data processing part we will use the torchtext module from PyTorch. The torchtext has utilities for creating datasets that can be easily iterated for the purposes of creating a language translation model. The below code will download the dataset and also tokenizes a raw text, build the vocabulary, and convert tokens into a tensor.

import math import torchtext import torch import chúng tôi as nn from torchtext.data.utils import get_tokenizer from collections import Counter from torchtext.vocab import Vocab from torchtext.utils import download_from_url, extract_archive from chúng tôi import pad_sequence from chúng tôi import DataLoader from torch import Tensor from chúng tôi import (TransformerEncoder, TransformerDecoder,TransformerEncoderLayer, TransformerDecoderLayer) import io import time

train_urls = (‘train.de.gz’, ‘train.en.gz’) val_urls = (‘val.de.gz’, ‘val.en.gz’) train_filepaths = [extract_archive(download_from_url(url_base + url))[0] for url in train_urls] val_filepaths = [extract_archive(download_from_url(url_base + url))[0] for url in val_urls] test_filepaths = [extract_archive(download_from_url(url_base + url))[0] for url in test_urls] de_tokenizer = get_tokenizer(‘spacy’, language=’de_core_news_sm’) en_tokenizer = get_tokenizer(‘spacy’, language=’en_core_web_sm’) def build_vocab(filepath, tokenizer): counter = Counter() with io.open(filepath, encoding=”utf8″) as f: for string_ in f: counter.update(tokenizer(string_)) de_vocab = build_vocab(train_filepaths[0], de_tokenizer) en_vocab = build_vocab(train_filepaths[1], en_tokenizer) def data_process(filepaths): raw_de_iter = iter(io.open(filepaths[0], encoding=”utf8″)) raw_en_iter = iter(io.open(filepaths[1], encoding=”utf8″)) data = [] for (raw_de, raw_en) in zip(raw_de_iter, raw_en_iter): de_tensor_ = torch.tensor([de_vocab[token] for token in de_tokenizer(raw_de.rstrip(“n”))], dtype=torch.long) en_tensor_ = torch.tensor([en_vocab[token] for token in en_tokenizer(raw_en.rstrip(“n”))], dtype=torch.long) data.append((de_tensor_, en_tensor_)) return data train_data = data_process(train_filepaths) val_data = data_process(val_filepaths) test_data = data_process(test_filepaths) device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’) BATCH_SIZE = 128

Then we will use the PyTorch DataLoader module which combines a dataset and a sampler, and it enables us to iterate over the given dataset. The DataLoader supports both iterable-style and map-style datasets with single or multi-process loading, also we can customize loading order and memory pinning.

# DataLoader def generate_batch(data_batch): de_batch, en_batch = [], [] for (de_item, en_item) in data_batch: de_batch.append(torch.cat([torch.tensor([BOS_IDX]), de_item, torch.tensor([EOS_IDX])], dim=0)) en_batch.append(torch.cat([torch.tensor([BOS_IDX]), en_item, torch.tensor([EOS_IDX])], dim=0)) de_batch = pad_sequence(de_batch, padding_value=PAD_IDX) en_batch = pad_sequence(en_batch, padding_value=PAD_IDX) return de_batch, en_batch train_iter = DataLoader(train_data, batch_size=BATCH_SIZE, shuffle=True, collate_fn=generate_batch) valid_iter = DataLoader(val_data, batch_size=BATCH_SIZE, shuffle=True, collate_fn=generate_batch) test_iter = DataLoader(test_data, batch_size=BATCH_SIZE, shuffle=True, collate_fn=generate_batch)

Then we are designing the transformer. Here the Encoder processes the input sequence by propagating it through a series of Multi-head Attention and Feedforward network layers. The output from this Encoder is referred to as memory below and is fed to the decoder along with target tensors. Encoder and decoder are trained in an end-to-end fashion.

# transformer class Seq2SeqTransformer(nn.Module): def __init__(self, num_encoder_layers: int, num_decoder_layers: int, emb_size: int, src_vocab_size: int, tgt_vocab_size: int, dim_feedforward:int = 512, dropout:float = 0.1): super(Seq2SeqTransformer, self).__init__() encoder_layer = TransformerEncoderLayer(d_model=emb_size, nhead=NHEAD, dim_feedforward=dim_feedforward) self.transformer_encoder = TransformerEncoder(encoder_layer, num_layers=num_encoder_layers) decoder_layer = TransformerDecoderLayer(d_model=emb_size, nhead=NHEAD, dim_feedforward=dim_feedforward) self.transformer_decoder = TransformerDecoder(decoder_layer, num_layers=num_decoder_layers) self.generator = nn.Linear(emb_size, tgt_vocab_size) self.src_tok_emb = TokenEmbedding(src_vocab_size, emb_size) self.tgt_tok_emb = TokenEmbedding(tgt_vocab_size, emb_size) self.positional_encoding = PositionalEncoding(emb_size, dropout=dropout) def forward(self, src: Tensor, trg: Tensor, src_mask: Tensor, tgt_mask: Tensor, src_padding_mask: Tensor, tgt_padding_mask: Tensor, memory_key_padding_mask: Tensor): src_emb = self.positional_encoding(self.src_tok_emb(src)) tgt_emb = self.positional_encoding(self.tgt_tok_emb(trg)) memory = self.transformer_encoder(src_emb, src_mask, src_padding_mask) outs = self.transformer_decoder(tgt_emb, memory, tgt_mask, None, tgt_padding_mask, memory_key_padding_mask) return self.generator(outs) def encode(self, src: Tensor, src_mask: Tensor): return self.transformer_encoder(self.positional_encoding( self.src_tok_emb(src)), src_mask) def decode(self, tgt: Tensor, memory: Tensor, tgt_mask: Tensor): return self.transformer_decoder(self.positional_encoding( self.tgt_tok_emb(tgt)), memory, tgt_mask)

The Text which is converted to tokens is represented by using token embeddings. The Positional encoding function is added to the token embedding so that we can get the notions of word order.

class PositionalEncoding(nn.Module): def __init__(self, emb_size: int, dropout, maxlen: int = 5000): super(PositionalEncoding, self).__init__() den = torch.exp(- torch.arange(0, emb_size, 2) * math.log(10000) / emb_size) pos = torch.arange(0, maxlen).reshape(maxlen, 1) pos_embedding = torch.zeros((maxlen, emb_size)) pos_embedding[:, 0::2] = torch.sin(pos * den) pos_embedding[:, 1::2] = torch.cos(pos * den) pos_embedding = pos_embedding.unsqueeze(-2) self.dropout = nn.Dropout(dropout) self.register_buffer('pos_embedding', pos_embedding) def forward(self, token_embedding: Tensor): return self.dropout(token_embedding + self.pos_embedding[:token_embedding.size(0),:]) class TokenEmbedding(nn.Module): def __init__(self, vocab_size: int, emb_size): super(TokenEmbedding, self).__init__() self.embedding = nn.Embedding(vocab_size, emb_size) self.emb_size = emb_size def forward(self, tokens: Tensor): return self.embedding(tokens.long()) * math.sqrt(self.emb_size)

Here in the below code, a subsequent word mask is created to stop a target word from attending to its subsequent words. Here the masks are also created, for masking source and target padding tokens.

def generate_square_subsequent_mask(sz): mask = (torch.triu(torch.ones((sz, sz), device=DEVICE)) == 1).transpose(0, 1) mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0)) return mask def create_mask(src, tgt): src_seq_len = src.shape[0] tgt_seq_len = tgt.shape[0] tgt_mask = generate_square_subsequent_mask(tgt_seq_len) src_mask = torch.zeros((src_seq_len, src_seq_len), device=DEVICE).type(torch.bool) src_padding_mask = (src == PAD_IDX).transpose(0, 1) tgt_padding_mask = (tgt == PAD_IDX).transpose(0, 1) return src_mask, tgt_mask, src_padding_mask, tgt_padding_mask

Then define the model parameters and instantiate the model.

SRC_VOCAB_SIZE = len(de_vocab) TGT_VOCAB_SIZE = len(en_vocab) EMB_SIZE = 512 NHEAD = 8 FFN_HID_DIM = 512 BATCH_SIZE = 128 NUM_ENCODER_LAYERS = 3 NUM_DECODER_LAYERS = 3 NUM_EPOCHS = 50 DEVICE = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu') transformer = Seq2SeqTransformer(NUM_ENCODER_LAYERS, NUM_DECODER_LAYERS, EMB_SIZE, SRC_VOCAB_SIZE, TGT_VOCAB_SIZE, FFN_HID_DIM) for p in transformer.parameters(): nn.init.xavier_uniform_(p) transformer = transformer.to(device) loss_fn = torch.nn.CrossEntropyLoss(ignore_index=PAD_IDX) optimizer = torch.optim.Adam( transformer.parameters(), lr=0.0001, betas=(0.9, 0.98), eps=1e-9 )

Define two different functions, that is for train and evaluation.

def train_epoch(model, train_iter, optimizer): model.train() losses = 0 for idx, (src, tgt) in enumerate(train_iter): src = src.to(device) tgt = tgt.to(device) tgt_input = tgt[:-1, :] src_mask, tgt_mask, src_padding_mask, tgt_padding_mask = create_mask(src, tgt_input) logits = model(src, tgt_input, src_mask, tgt_mask, src_padding_mask, tgt_padding_mask, src_padding_mask) optimizer.zero_grad() tgt_out = tgt[1:, :] loss = loss_fn(logits.reshape(-1, logits.shape[-1]), tgt_out.reshape(-1)) loss.backward() optimizer.step() losses += loss.item() torch.save(model, PATH) return losses / len(train_iter) def evaluate(model, val_iter): model.eval() losses = 0 for idx, (src, tgt) in (enumerate(valid_iter)): src = src.to(device) tgt = tgt.to(device) tgt_input = tgt[:-1, :] src_mask, tgt_mask, src_padding_mask, tgt_padding_mask = create_mask(src, tgt_input) logits = model(src, tgt_input, src_mask, tgt_mask, src_padding_mask, tgt_padding_mask, src_padding_mask) tgt_out = tgt[1:, :] loss = loss_fn(logits.reshape(-1, logits.shape[-1]), tgt_out.reshape(-1)) losses += loss.item() return losses / len(val_iter) Now training the model. for epoch in range(1, NUM_EPOCHS+1): start_time = time.time() train_loss = train_epoch(transformer, train_iter, optimizer) end_time = time.time() val_loss = evaluate(transformer, valid_iter) print((f"Epoch: {epoch}, Train loss: {train_loss:.3f}, Val loss: {val_loss:.3f}, " f"Epoch time = {(end_time - start_time):.3f}s"))

This model is trained using transformer architecture in such a way that it trains faster and also it converges to a lower validation loss compared to other RNN models.

def greedy_decode(model, src, src_mask, max_len, start_symbol): src = src.to(device) src_mask = src_mask.to(device) memory = model.encode(src, src_mask) ys = torch.ones(1, 1).fill_(start_symbol).type(torch.long).to(device) for i in range(max_len-1): memory = memory.to(device) memory_mask = torch.zeros(ys.shape[0], memory.shape[0]).to(device).type(torch.bool) tgt_mask = (generate_square_subsequent_mask(ys.size(0)) .type(torch.bool)).to(device) out = model.decode(ys, memory, tgt_mask) out = out.transpose(0, 1) prob = model.generator(out[:, -1]) _, next_word = torch.max(prob, dim = 1) next_word = next_word.item() ys = torch.cat([ys, torch.ones(1, 1).type_as(src.data).fill_(next_word)], dim=0) if next_word == EOS_IDX: break return ys def translate(model, src, src_vocab, tgt_vocab, src_tokenizer): model.eval() tokens = [BOS_IDX] + [src_vocab.stoi[tok] for tok in src_tokenizer(src)] + [EOS_IDX] num_tokens = len(tokens) src = (torch.LongTensor(tokens).reshape(num_tokens, 1)) src_mask = (torch.zeros(num_tokens, num_tokens)).type(torch.bool) tgt_tokens = greedy_decode(model, src, src_mask, max_len=num_tokens + 5, start_symbol=BOS_IDX).flatten() Now, let’s test our model on translation. output = translate(transformer, "Eine Gruppe von Menschen steht vor einem Iglu .", de_vocab, en_vocab, de_tokenizer) print(output)

Above the red line is the output from the translation model. You can also compare it with google translator.

Source: Google Translator

The above translation and the output from our model matched. The model is not the best but still does the job up to some extent.

Reference

Thank you

The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.

You're reading Language Translation With Transformer In Python!

C++ Vs Python: Which Programming Language Will Takeover Tiobe Index In 2023?

C++ vs Python: C++ is now aiming for Python’s first place on the Tiobe index in 2023

C++ vs Python are two different languages that have different features and different behavior. Both these languages have one thing in common i.e., strong support for object-oriented programming. This article features the difference between two world-famous programming languages, C++ and Python, and also we will discuss which programming language will take over Tiobe Index in 2023.

C++ vs Python: Which Programming Language Will Take over Tiobe Index?

First, let us dive into the efficiencies and effectiveness of Python. It is a high-level language and one of the top programming languages that are taught and used by institutes around the world. Developers can build robust applications with the help of Python due to its English-like syntax. There are several quantum professionals without any prior exposure to coding or programming, hence, using Python programming might ease up their doubts and guide them toward executing complex problems quite quickly. The primary characteristic of Python that makes it so desirable among developers and coders is its accessibility. Python is absolutely free to use and anybody can use and download the source code, modify it, and conduct programming and coding capabilities, without any prior training.

C++ on the other hand is a high-level, general-purpose programming language created by Bjarne Stroustrup in 1979 as an extension of the C programming language, or “C with Classes”. The concept of object-oriented programming was first introduced in the C++ language. C++ is also known as an object-oriented programming language. It was designed for system programming and embedded systems, but later on, it was used in developing various applications such as desktop applications, video games, servers such as e-commerce, Web search or SQL servers, and performance-critical applications such as telephone switches. Many of the technologies as libraries in Python have underlying C++ code.

C++ is faster than a python programming language. Python is written in the C programming language, so memory management is very difficult in python. In C++, we can allocate the memory to the variables and can deallocate the memory when the variable is no longer used in the code.

According to the Tiobe programming languages index, C++ has already overtaken Java in popularity and is now aiming for Python as it is in first place on the Tiobe index. Java is currently ranked fourth in the TIOBE index after C++ overtook Java for the first time.

The Tiobe programming language index indicates that Python is in the first position. According to Paul Jansen, CEO of Tiobe Software, a Dutch company that performs software quality testing, this is the first time C++ has surpassed Java in the Tiobe index and it’s the first time since 2001 that Java hasn’t been in the top three.

What is the Tiobe index?

The Netherlands-based TIOBE Software BV, which is based in Eindhoven, devised and maintains the TIOBE programming community index as a metric for programming language popularity. The index is determined by counting how many times the name of the language appears in search engine results. The index includes Wikipedia, YouTube, MSN, Yahoo!, Baidu, Google Blogs, and Google. Once every month, the index is updated. The long-term statistical data is for sale, while the current information is available for free. The index creators claim that it might be helpful while deciding on various strategic options. TIOBE does not offer data on, for example, the popularity of HTML because it concentrates on Turing complete languages.

C++ vs Python: Which one is better to learn?

It ideally depends on the programmer as to what he/she wants to learn. Secondly, it also depends on the current requirements. Suppose you want to learn system programming or any such low-level programming, C++ is your ideal choice.

But if you want some machine learning knowledge and put it to practice, then you should go for Python. Alternatively, if you want to make yourself comfortable in web programming, then you could opt for Ruby or JavaScript or angular JS, etc.

Why Python Is The Best Programming Language For Microservices?

Microservices is a service-oriented architecture pattern wherein applications are built as a collection of various smallest independent service units.

Python is a high-level programming language that offers active support for integration with various technologies. Python is commonly used for developing websites and software, task automation, data analysis, and data visualization. Since it’s relatively easy to learn, Python has been adopted by many non-programmers such as accountants and scientists, for a variety of everyday tasks, like organizing finances. Prototyping in Python is faster and easier when compared to other frameworks and programming languages. It includes powerful substitutes for heavy implementations like Django. Microservices Python ensures compatibility with legacy languages like ASP and PHP, which allow you to create web service front-ends to host Microservices.

With all these benefits, Microservices Python is considered to have an edge over other languages. Developers who implement Microservices Python use a RESTful API approach – a comprehensive way of utilizing web protocols & software to remotely manipulate objects. With this technology, it becomes easier to monitor the application since it is now broken into components. There is a broad range of Python microservices frameworks to choose from for your web application development. Some of them are as follows:

Flask – Most popular Python Micro framework based on Jinja2 and Werkzeug

Falcom – Create smart proxies, cloud APIs, and app back-ends

Bottle – Simple, lightweight, and fast WSGI micro framework

Nameko – Best among the Python Microservices frameworks that allow developers to concentrate on application logic

CherryPy – Mature, Python object-oriented web framework

What is microservice architecture?

It is a software development approach that is used to disintegrate mobile applications into smaller parts. Microservice architecture is rapidly replacing monolithic architecture that is used in heavier, complex applications.

The basic focus of the microservice architecture is to develop cloud-ready apps and simplify the deployment process. The architecture has several built-in programming languages and also uses different data storage techniques.

Avoid the potholes

Thinking of migrating to the microservice architecture? If so, you should look at this presentation about the potholes in the road from monolithic hell and read this series of blog posts about anti-patterns and ways to avoid them.

Assess your architecture

If you have built an application with the microservice architecture then take a look at the Microservices Assessment Platform. The platform assesses what you have built and identifies what needs to be improved. It reduces architectural and organizational risk and maximizes the benefits of microservice architecture.

The process to choose programming languages for microservices

Organizations need to understand that microservices can be implemented with a plethora of frameworks and tools. Hence, it is necessary to employ the best practices while choosing a programming language for microservices. Here are some of the criteria that will help in evaluating the best programming language for microservices.

The language must be independent of deployment and must be highly observable.

It must have a customer-centric approach and according to the changing trends, must support automation.

The structure of the language should be around the business domain.

It must have decentralization of components and must support continuous integration.

Python Microservice Monitoring With Interceptors

Once you have some microservices in the cloud, you want to have visibility into how they’re doing. Some things you want to monitor include:

How many requests each microservice is getting

How many requests result in an error, and what type of error they raise

The latency on each request

Exception logs so you can debug later

Why Is Python The Best Language For Web Scraping?

What is Python Web Scraping?

Python Web Scraping is an automatic method of collecting data from the web and its different websites, and performing further operations on the data. These may include storing the data in a database for future references, analyzing the data for business purposes, and providing a continuous stream of data from different sources in a single place.

Some common methods of web scraping

High performance

Simple syntax

Available existing frameworks

Universality of Python

Useful data representation

Let us take a detailed look.

Reason 1: High Performance

Python scripts written for web scraping are highly efficient. Web scraping is limited to just retrieving data from other sources in some languages, whereas in some others, it involves sourcing the data in an unstructured format and appending it together, followed by parsing and saving it as a dataset. Scripts written in Python do all this, as well as representing the scraped data visually with Python libraries like Matplotlib.

Syntax tree = html.fromstring(response.text) text_in_site = tree.xpath(‘IP address/text()’) for title in blog_titles: print(title)

Here we are seeing a scraping script using the lxml library of Python. This library contains a html module to work with HTML, although it needs the HTML string first which is retrieved using the Requests library. This parsed data is then stored in a tree object, and exact data items can be accessed by creating queries using the Xpath() function, from which desired components like text or body of the website can be extracted using appropriate tags.

Algorithm

Step 1 − Import the lxml library

Step 2 − Retrieve the HTML string using Requests library

Step 3 − Parse the scraped data from target website

Step 4 − Obtain individual data elements by using queries

Step 5 − Printing the required data, or using it for further purposes

Example # After response = requests.get() from lxml import html tree = html.fromstring(response.text) blog_titles=tree.xpath('//h2[@class="blog-card__content-title"]/text()') for title in blog_titles: print(title)

This script only runs in dedicated Python IDEs such as Jupyter Notebook/terminals.

Output Blog title 1 Blog title 2 Blog title 3 Reason 2: Simple Syntax

The Python language has one of the easiest and most simple syntaxes of the programming world. This is what makes it one of the easiest languages to learn for beginners. Thus, web scraping scripts written in Python are very small and simple, compared to other languages like C# and C++. This is what makes web scraping using Python so easy to write and execute.

Syntax pip install requests import requests print(response.text)

Here we use the Requests library to perform web scraping, which has one of the easiest and shortest code scripts to execute. The library sends a HTTP request using the GET() function, and then prints the scraped data for the user. This can be used as the basic syntax for the Requests library and can be modified as needed.

Algorithm

Step 1 − Install the Requests library using the console

Step 2 − Send HTTP request to the website server using the REQUESTS.GET() command

Step 3 − Print the received scraped data or use it for necessary representation purposes.

Example import requests from bs4 import BeautifulSoup print("n") soup_data = BeautifulSoup(res.text, 'html.parser') print(soup_data.title) print("n") print(soup_data.find_all('h4'))

This script only runs in dedicated Python IDEs such as Jupyter Notebook/terminals.

Output [Academic, Computer Science, Digital Marketing, Monuments, Machine Learning, Mathematics, Mobile Development, SAP, Software Quality, Big Data & Analytics, Databases, Engineering Tutorials, Mainframe Development, Microsoft Technologies, Java Technologies, XML Technologies, Python Technologies, Sports, Computer Programming, DevOps, Latest Technologies, Telecom, Exams Syllabus, UPSC IAS Exams, Web Development, Scripts, Management, Soft Skills, Selected Reading, Misc] Reason 3: Available Existing Frameworks

The Python language has an extensive collection of frameworks for a wide range of functions and use cases, which includes web scraping as well. Libraries such as Beautiful Soup, lxml, Requests and Scrapy. Using these frameworks for web scraping can be very efficient and effective, and can also support Xpath, HTML and other. These libraries also contain debugging methods, which help in smooth and secure programming.

Syntax driver = Chrome(executable_path='/path/to/driver')

Here we are using Selenium for web scraping, which supports parsing using Javascript, thereby allowing crawling on dynamic websites. Here we require a driver for the browser being used. In today’s era of the entire internet being programmed on Javascript, this library is essential for web scraping.

Algorithm

Step 1 − Installing the Selenium library

Step 2 − Importing the appropriate class for the browser used

Step 3 − Object of the browser is created using the driver

Step 4 − Load the required webpage using the get() method

Step 5 − Extract the neccessary elements from the website, if required

Step 6 − Close the browser object

Example from selenium import webdriver from selenium.webdriver.chrome.options import Options options = Options() options.headless = True options.add_argument("--window-size=1920,1200") DRIVER_PATH = '/path/to/chromedriver' driver = webdriver.Chrome(options=options, executable_path=DRIVER_PATH) print(driver.page_source) driver.quit()

This script only runs in dedicated Python IDEs such as Jupyter Notebook/terminals.

Reason 4: Universality of Python

Python is one of the most universally used programming languages in today’s world, and is also widely accepted in different aspects. The biggest data collectors and companies in the world use Python, and scripts written in Python can be used with programs written in other languages as well.

Syntax pip import requests import requests print(response.text)

Here we use a web scraping script using the Requests library, which can be used in sync with scripts written in other languages and programming environments as well, thereby making Python scripts universal.

Algorithm

Step 1 − Install the Requests library using the console

Step 2 − Send HTTP request to the website server using the REQUESTS.GET() command

Step 3 − Print the received scraped data or use it for necessary representation purposes.

Example pip import requests import requests print(response.text)

This script only runs in dedicated Python IDEs such as Jupyter Notebook/terminals.

Output Reason 5: Useful Data Representation

The web scraping libraries used in Python can perform not just web crawling and data parsing, they can execute useful representation of data for purposes like business analysis, research and market analysis, and understanding customer feedback. Beautiful Soup is the best for scraping data which can be then displayed via Matplotlib, Plotly and similar libraries.

Syntax response = requests.get(url) soup = BeautifulSoup(response.text, 'html.parser')

This is the syntax for a script in Beautiful Soup, where we first get the target url using the Requests library, as shown in earlier examples. Then we search and find the required element from the website in the second line. This received data can be then represented using the appropriate libraries.

Algorithm

Step 1 − Install the Beautiful Soup library

Step 2 − Receive the website url by sending request

Step 3 − Extract the element required from the website

Step 4 − Perform necessary operations with the data like printing/storing etc.

Step 5 − Pass the data to Matplotlib for representation purposes

Example import requests response = requests.get(url) from bs4 import BeautifulSoup soup = BeautifulSoup(response.text, 'html.parser') print(soup.title) blog_titles = soup.select('h2.blog-card__content-title') for title in blog_titles: print(title.text)

This script only runs in dedicated Python IDEs such as Jupyter Notebook/terminals.

Output Conclusion

Thus here we see how web scraping is done using various methods in Python, and also the way in which these methods make Python the best for web scraping. There are several other smaller reasons why Python is great for web scraping, but here we have only mentioned a few. To get a detailed lesson on each of the methods, you can visit their individual learning pages respectively. Python is thus arguably one of the best languages to perform web scraping.

Import Module In Python With Examples

What are the modules in Python?

A module is a file with python code. The code can be in the form of variables, functions, or class defined. The filename becomes the module name.

For example, if your filename is chúng tôi the module name will be guru99. With module functionality, you can break your code into different files instead of writing everything inside one file.

In this tutorial, you will learn:

What is the Python import module?

A file is considered as a module in python. To use the module, you have to import it using the import keyword. The function or variables present inside the file can be used in another file by importing the module. This functionality is available in other languages, like typescript, JavaScript, java, ruby, etc.

How to create and import a module in Python?

Now we will create a module and import it in another file.

Here is the flow to create and import the module as shown in the screenshot:

Follow the steps given to create a module in python.

The folder structure used to test the code is as follows:

modtest/ test.py display.py

Step 1) Create a file and name it test.py

Step 2) Inside chúng tôi create a function called display_message()

Def display_message(): return "Welcome to Guru99 Tutorials!"

Step 3) Now create another file display.py.

Step 4) Inside chúng tôi import the chúng tôi file, as shown below:

import test

While importing, you don’t have to mention the chúng tôi but just the name of the file.

Step5)

Then you can call the function display_message() from chúng tôi inside chúng tôi you need to make use of module_name.function_name.

For example test.display_message().

Import test print(test.display_message())

Step 6)

When you execute chúng tôi you will get the following output:

Welcome to Guru99 Tutorials! Importing a Class in Python

Earlier, we have seen a simple module with a function. Here will create a class and refer the class inside another file.

The folder structure to test the code is as follows:

myproj/ Car.py display.py

Create a file called chúng tôi with the following code:

Filename : Car.py

class Car: brand_name = "BMW" model = "Z4" manu_year = "2023" def __init__(self, brand_name, model, manu_year): self.brand_name = brand_name self.model = model self.manu_year = manu_year def car_details(self): print("Car brand is ", self.brand_name) print("Car model is ", self.model) print("Car manufacture year is ", self.manu_year) def get_Car_brand(self): print("Car brand is ", self.brand_name) def get_Car_model(self): print("Car model is ", self.model)

In the file chúng tôi , there are attributes brand_name, model and manu_year. The functions defined inside the class are car_details(), get_Car_brand(), get_Car_model().

Let us now use the file chúng tôi as a module in another file called display.py.

Filename : display.py

import Car car_det = Car.Car("BMW","Z5", 2023) print(car_det.brand_name) print(car_det.car_details()) print(car_det.get_Car_brand()) print(car_det.get_Car_model())

Output:

BMW Car brand is BMW Car model is Z5 Car manufacture year is 2023 Car brand is BMW Car model is Z5

So we can access all the variables and functions from chúng tôi using Car module.

Using from to import module

You can import only a small part of the module i.e., only the required functions and variable names from the module instead of importing full code.

When you want only specific things to be imported, you can make use of “from” keyword to import what you want.

So the syntax is

from module import your function_name , variables,... etc.

The folder structure used to test the code is as follows:

modtest/ test.py display.py

In chúng tôi there are 2 functions as shown :

Filename : test.py

defdisplay_message(): return "Welcome to Guru99 Tutorials!" def display_message1(): return "All about Python!"

Now you want display_message() function. The function or variable that you are importing can be directly accessed as shown below:

File Name : display.py

from test import display_message print(display_message())

Output:

Welcome to Guru99 Tutorials!

Now if you happen to use the function display_message1() ,it will throw an error that the function is not defined as shown below:

from test import display_message print(display_message1())

Output:

Traceback (most recent call last): print(display_message1()) Name Error: name 'display_message1' is not defined Importing everything from the module

Import allows you to import the full module by using import followed by module name, i.e., the filename or the library to be used.

Syntax:

Import module

Or by using

from module import *

The folder structure used to test the code is as follows:

modtest/ test.py display.py

Following are the code details inside test.py

my_name = "Guru99" my_address = "Mumbai" defdisplay_message(): return "Welcome to Guru99 Tutorials!" def display_message1(): return "All about Python!" Using the import module

Using just import module name, to refer to the variables and functions inside the module, it has to prefix with module name.

Example

Filename : display.py

Import test print(test.display_message()) print(test.display_message1()) print(test.my_name) print(test.my_address)

The module name test is used to refer to the function and variables inside module test.

Output:

Welcome to Guru99 Tutorials! All about Python! Guru99 Mumbai Using import *

Let us see an example using import *. Using import *, the functions and variables are directly accessible, as shown in the example below:

from test import * print(display_message()) print(display_message1()) print(my_name) print(my_address)

Output:

Welcome to Guru99 Tutorials! All about Python! Guru99 Mumbai The dir( ) function

The dir() is a built-in-function in python. The dir() returns all the properties and methods, including the given object’s built-in properties.

So when dir() is used on the module, it will give you the variables, functions that are present inside the module.

Here is a working example of dir() on a module. We have a class called chúng tôi let us import Car and assign to dir() to see the output.

The folder structure to test the code will be as follows:

test prop/ Car.py test.py

Filename: Car.py

Filename: test.py

import Car class_contents = dir(Car) print(class_contents)

The output gives us the name of the class and all the functions defined in Car.py.

You can also try using dir() on a built-in module available in Python. Let us try the same on json module as shown in the example below. It will display all the properties and methods available in json module.

Import json json_details = dir(json) print(json_details)

Output:

['JSONDecodeError', 'JSONDecoder', 'JSONEncoder', '__all__', '__author__', '__bu iltins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__pac kage__', '__path__', '__spec__', '__version__', '_default_decoder', '_default_en coder', 'codecs', 'decoder', 'detect_encoding', 'dump', 'dumps', 'encoder', 'loa Packages

A package is a directory with all modules defined inside it. To make a Python interpreter treat it as a package, your directory should have the init.pyfile. The chúng tôi makes the directory as a package. Here is the layout of the package that we are going to work on.

The name of the package is my package. To start working with the package, create a directory called package/. Inside the directory, create an empty file called __init__.py. Create 3 more files chúng tôi chúng tôi and chúng tôi and define the functions as shown in the screenshot. Here are the details of module1.py,module2.py and module3.py

chúng tôi

def mod1_func1(): print("Welcome to Module1 function1") def mod1_func2(): print("Welcome to Module1 function2") def mod1_func3(): print("Welcome to Module1 function3")

chúng tôi

def mod2_func1(): print("Welcome to Module2 function1") def mod2_func2(): print("Welcome to Module2 function2") def mod2_func3(): print("Welcome to Module2 function3")

chúng tôi

def mod3_func1(): print("Welcome to Module3 function1") def mod3_func2(): print("Welcome to Module3 function2") def mod3_func3(): print("Welcome to Module3 function3")

The packages ready for use. Now call the package inside any of your file as shown below :test.py:

Here, the mypackage.module1 is imported and given an alias name as mod1. Similarly, you can use other modules chúng tôi and chúng tôi from my package.

import mypackage.module1 as mod1 print(mod1.mod1_func1()) print(mod1.mod1_func2()) print(mod1.mod1_func2())

Output:

Welcome to Module1 function1 None Welcome to Module1 function2 None Welcome to Module1 function2 None

We have just demonstrated the package with a simple module with functions inside it. As per your project, you can also package with have sub-packages. Sub-folders/ having modules with classes defined.

Python Module Search Path

During execution, when python comes across import module name, the interpreter tries to locate the module. It searches the module in the build-in module list. Later in all, the directories defined inside sys.path.

To sum up, the interpreter does the following search to locate the module:

In your current directory.

In in the build-in module list

Inside the chúng tôi directories

You can get the details of chúng tôi by importing sys module and print the sys.path. It will give you the list of directories as shown below:

importsys print(sys.path)

Output:

['Python Latest\task2', 'Users\AppData\Local\Programs\Python Python37\python37.zip', 'Users\AppData\Local\Programs\Python\P ython37\DLLs']

You can also modify the path and keep the directories as per your requirements.

Using module alias in the import

You can also convert the module name to a shorter form by giving an alias name to it. The alias can be done using the keyword.

Syntax:

import filename as alias name

The folder structure to test the code will be as follows:

Mod test/ test.py display.py

Following is the code inside test.py

my_name = "Guru99" my_address = "Mumbai" def display_message(): return "Welcome to Guru99 Tutorials!" def display_message1(): return "All about Python!"

Now will use an alias for chúng tôi in display.py

Import test as t print(t.display_message()) print(t.display_message1()) print(t.my_name) print(t.my_address)

The alias that is used for test module is t . So the function and variables from chúng tôi can be referred using the alias t.

Output:

Welcome to Guru99 Tutorials! All about Python! Guru99 Mumbai Absolute and Relative Imports in Python

You now know how to import a file as a module inside another file. Let us now see how to manage the files available in folders. The files in the folders can be imported either by using absolute or relative imports.

Consider you have your project folder structure, as shown below:

The root folder is my project/. It has two subfolders package1 and package2.

The folder package1 has two modules, chúng tôi and module2.py.

The folder package2 has one class chúng tôi a sub-package subpkg with chúng tôi and last module4.py.

In chúng tôi there is a functioncalledmyfunc1.

In chúng tôi there is a functioncalledmyfunc2.

In chúng tôi there is a functioncalledmyfunc3.

In chúng tôi there is a functioncalledmyfunc4.

Using Absolute Imports

For Absolute imports, you need to add the entire path of your module right from the project root folder.

Let us now see how to make use of absolute imports to refer to the functions present in each of the module.

To work with the functionmyfunc1, you will need to import as follows:

from package1.module1 import myfunc1 or from package1 import module1 module1.myfunc1()

To work with the function myfunc3 you will need to import as follows:

from package1.subpkg.module3 import myfunc3 or from package1.subpkg import module3 module3.myfunc3()

It becomes easy to trace back the modules for code check.

Easy to use and very straightforward.

If the project is moved to a different path, still the imports will remain the same.

The import path can get very long in-case, the modules are nested, and if the name of the modules is lengthy.

Using Relative Imports

Considering the same folder structure mentioned below, we will see how to import the same using relative imports.

In relative import, the module to be imported is relative to the current location that is the location where the import statement is present.

Syntax:

In relative imports, you need to add a period (.) before the module name when importing using from.

It will be 2 periods (..) before the module name if the module is in the one level up from the current location.

Referring to the folder structure figure mentioned above, we have the following modules with their function, which we need to refer to.

In chúng tôi there is a functioncalledmyfunc1.

In chúng tôi there is a functioncalledmyfunc2.

In chúng tôi there is a functioncalledmyfunc3.

In chúng tôi there is a functioncalledmyfunc4.

To work with the functionmyfunc1 you will need to import as follows:

from .module1 import myfunc1

To work with the function myfunc3, you will need to import as follows:

from .subpkg.module3 import myfunc3 Advantages of Relative Imports

Advantages:

It is easy to work with relative imports.

From the current location, the imports can be shortened in comparison to absolute imports.

Using relative imports, it is difficult to trace back where the code resides

Summary:

Import in Python helps you to refer to the code, i.e., .functions/objects that are written in another file. It is also used to import python libraries/packages that are installed using pip(python package manager), and you need then to use in your code.

Import functionality is available in other languages like typescript, JavaScript, java, ruby, etc.

A module is python is the code written inside the file, for example (test.py). Inside your file, you can have your variables, functions, or your class defined. The entire file becomes a module and can be imported inside another file to refer to the code.

With module functionality, you can break your code into different files instead of writing everything inside one file. Later, using import, you can refer to the code inside the file you need.

Python has its built-in modules, and also external libraries/packages installed using a python package manager (pip), e.g., pandas, NumPy, etc. are referred to as modules.

You can import only a small part of the module, i.e., only the required functions and variable names from the module instead of importing full code.

You can also convert the module name to a shorter form by giving an alias name to it. The alias can be done using the keyword.

A package is a directory with all modules defined inside it. To make a Python interpreter treat it as a package, your directory should have the __init.pyfile. The chúng tôi makes the directory as a package. Here is the layout of the package that we are going to work on.

For Absolute imports, you need to add the entire path of your module right from the project root folder.

In relative import, the module to be imported is relative to the current location that is the location where the import statement is present.

Transformer Coupled Class A Power Amplifier

The construction of class A power amplifier can be understood with the help of below figure. This is similar to the normal amplifier circuit but connected with a transformer in the collector load.

Here R1 and R2 provide potential divider arrangement. The resistor Re provides stabilization, Ce is the bypass capacitor and Re to prevent a.c. voltage. The transformer used here is a step-down transformer.

The high impedance primary of the transformer is connected to the high impedance collector circuit. The low impedance secondary is connected to the load (generally loud speaker).

Transformer Action

The transformer used in the collector circuit is for impedance matching. RL is the load connected in the secondary of a transformer. RL’ is the reflected load in the primary of the transformer.

The number of turns in the primary are n1 and the secondary are n2. Let V1 and V2 be the primary and secondary voltages and I1 and I2 be the primary and secondary currents respectively. The below figure shows the transformer clearly.

We know that

$$frac{V_1}{V_2} = frac{n_1}{n_2}: and: frac{I_1}{I_2} = frac{n_1}{n_2}$$

$$V_1 = frac{n_1}{n_2}V_2 : and: I_1 = frac{n_1}{n_2}I_2$$

Hence

$$frac{V_1}{I_1} = left ( frac{n_1}{n_2} right )^2 frac{V_2}{I_2}$$

But V1/I1 = RL’ = effective input resistance

And V2/I2 = RL = effective output resistance

Therefore,

$$R_L’ = left ( frac{n_1}{n_2}right )^2 R_L = n^2 R_L$$

Where

$$n = frac{number : of : turns : in : primary}{number: of: turns: in: secondary} = frac{n_1}{n_2}$$

A power amplifier may be matched by taking proper turn ratio in step down transformer.

Circuit Operation

If the peak value of the collector current due to signal is equal to zero signal collector current, then the maximum a.c. power output is obtained. So, in order to achieve complete amplification, the operating point should lie at the center of the load line.

The operating point obviously varies when the signal is applied. The collector voltage varies in opposite phase to the collector current. The variation of collector voltage appears across the primary of the transformer.

Circuit Analysis

The power loss in the primary is assumed to be negligible, as its resistance is very small.

The input power under dc condition will be

$$(P_{in})_{dc} = (P_{tr})_{dc} = V_{CC} times (I_C)_Q$$

Under maximum capacity of class A amplifier, voltage swings from (Vce)max to zero and current from (Ic)max to zero.

Hence

$$V_{rms} = frac{1}{sqrt{2}} left [frac{(V_{ce})_{max} – (V_{ce})_{min}}{2} right ] = frac{1}{sqrt{2}} left[ frac{(V_{ce})_{max}}{2}right ] = frac{2V_{CC}}{2sqrt{2}} = frac{V_{CC}}{sqrt{2}}$$

$$I_{rms} = frac{1}{sqrt{2}} left [frac{(I_C)_{max} – (I_C)_{min}}{2} right ] = frac{1}{sqrt{2}} left[ frac{(I_C)_{max}}{2}right ] = frac{2(I_C)_Q}{2sqrt{2}} = frac{(I_C)_Q}{sqrt{2}}$$

Therefore,

$$(P_O)_{ac} = V_{rms} times I_{rms} = frac{V_{CC}}{sqrt{2}} times frac{(I_C)_Q}{sqrt{2}} = frac{V_{CC} times (I_C)_Q}{2}$$

Therefore,

Collector Efficiency = $frac{(P_O)_{ac}}{(P_{tr})_{dc}}$

Or,

$$(eta)_{collector} = frac{V_{CC} times (I_C)_Q}{2 times V_{CC} times (I_C)_Q} = frac{1}{2}$$

$$= frac{1}{2} times 100 = 50%$$

The efficiency of a class A power amplifier is nearly than 30% whereas it has got improved to 50% by using the transformer coupled class A power amplifier.

Advantages

No loss of signal power in the base or collector resistors.

Excellent impedance matching is achieved.

Gain is high.

DC isolation is provided.

Low frequency signals are less amplified comparatively.

Hum noise is introduced by transformers.

Transformers are bulky and costly.

Poor frequency response.

Applications

The applications of transformer coupled class A power amplifier are as follows.

This circuit is where impedance matching is the main criterion.

These are used as driver amplifiers and sometimes as output amplifiers.