Author: Antonio Gulli

  • loc 458 - If we have a big output jump, we cannot progressively learn (rather than trying things in all possible directions—a process known as exhaustive search—without knowing if we are improving).

  • loc 460 - A perceptron is either 0 or 1 and that is a big jump and it will not help it to learn, as shown in the following graph: We need something different, smoother. We need a function that progressively changes from 0 to 1 with no discontinuity.

  • loc 469 - A neuron can use the sigmoid for computing the nonlinear function . Note that, if  is very large and positive, then , so , while if  is very large and negative  so .

  • loc 476 - The sigmoid is not the only kind of smooth activation function used for neural networks. Recently, a very simple function called rectified linear unit (ReLU) became very popular because it generates very good experimental results. A ReLU is simply defined as , and the nonlinear function is represented in the following graph.

  • loc 476 - The sigmoid is not the only kind of smooth activation function used for neural networks. Recently, a very simple function called rectified linear unit (ReLU) became very popular because it generates very good experimental results. A ReLU is simply defined as , and the nonlinear function is represented in the following graph. As you can see in the following graph, the function is zero for negative

  • loc 476 - The sigmoid is not the only kind of smooth activation function used for neural networks. Recently, a very simple function called rectified linear unit (ReLU) became very popular because it generates very good experimental results. A ReLU is simply defined as , and the nonlinear function is represented in the following graph. As you can see in the following graph, the function is zero for negative values, and it grows linearly for positive values:

  • loc 504 - In many applications, it is convenient to transform categorical (non-numerical) features into numerical variables. For instance, the categorical feature digit with the value d in [0-9] can be encoded into a binary vector with 10 positions, which always has 0 value, except the d-th position where a 1 is present. This type of representation is called one-hot encoding (OHE) and is very common in data mining when the learning algorithm is specialized for dealing with numerical functions.

  • loc 554 - Binary cross-entropy: This is the binary logarithmic loss. Suppose that our model predicts p while the target is t, then the binary cross-entropy is defined as follows: This objective function is suitable for binary labels prediction.

  • loc 565 - Precision: This denotes how many selected items are relevant for a multilabel classification

  • loc 566 - Recall: This denotes how many selected items are relevant for a multilabel classification

  • loc 679 - Note that it has been frequently observed that networks with random dropout in internal hidden layers can generalize better on unseen examples contained in test sets. Intuitively, one can think of this as each neuron becoming more capable because it knows it cannot depend on its neighbors.

  • loc 785 - As a rule of thumb, if during the training we see that the loss increases on validation, after an initial decrease, then we have a problem of model complexity that overfits training. Indeed,

  • loc 789 - if we have two models, M1 and M2, achieving pretty much the same performance in terms of loss function, then we should choose the simplest model that has the minimum number of nonzero weights.

  • loc 791 - We can use a hyperparameter ⅄>=0 for controlling what the importance of having a simple model is, as in this formula:

  • loc 794 - L1 regularization (also known as lasso): The complexity of the model is expressed as the sum of the absolute values of the weights L2 regularization (also known as ridge): The complexity of the model is expressed as the sum of the squares of the weights Elastic net regularization: The complexity of the model is captured by a combination of the two preceding techniques

  • loc 1304 - There are two options: padding=‘valid’ means that the convolution is only computed where the input and the filter fully overlap, and therefore the output is smaller than the input, while padding=‘same’ means that we have an output that is the same size as the input, for which the area around the input is padded with zeros.

  • loc 2347 - In other words, distributed representation aims to convert words to vectors where the similarity between the vectors correlate with the semantic similarity between the words.

  • loc 2351 - The models are unsupervised, taking as input a large corpus of text and producing a vector space of words.

  • loc 2353 - The embedding space is also more dense compared to the sparse embedding of the one-hot embedding space.

  • loc 2355 - the CBOW architecture, the model predicts the current word given a window of surrounding words. In addition, the order of the context words does not influence the prediction (that is, the bag of words assumption).

  • loc 2358 - According to the authors, CBOW is faster but skip-gram does a better job at predicting infrequent words.

  • loc 2363 - love green eggs and ham.

  • loc 2366 - Since the skip-gram model predicts a context word given the center word, we can convert the preceding dataset to one of (input, output) pairs. That is, given an input word, we expect the skip-gram model to predict the output word: (love, I), (love, green), (green, love), (green, eggs), (eggs, green), (eggs, and), …