关于 Dolly 2.0

Hugging Face Optimum Intel API

下载及转换模型(注意：此模型大小约10GB)

-from transformers import AutoModelForCausalLM
+from optimum.intel.openvino import OVModelForCausalLM
from transformers import AutoTokenizer, pipeline

model_id = "databricks/dolly-v2-3b"
-model = AutoModelForCausalLM.from_pretrained(model_id)
+model = OVModelForCausalLM.from_pretrained(model_id, from_transformers=True)

模型类初始化从调用 from_pretrained开始。当下载及转换模型, 需添加此参数from_transformers=True。我们用save_pretrained储存转换模型。 Tokenizer class and pipelines API 是与Optimus 模型兼容的。

from pathlib import Path
from transformers import AutoTokenizer
from optimum.intel.openvino import OVModelForCausalLM

model_id = "databricks/dolly-v2-3b"
model_path = Path("dolly-v2-3b")

tokenizer = AutoTokenizer.from_pretrained(model_id)

current_device = "CPU"

if model_path.exists():
    ov_model = OVModelForCausalLM.from_pretrained(model_path, device=current_device)
else:
    ov_model = OVModelForCausalLM.from_pretrained(model_id, device=current_device, from_transformers=True)
    ov_model.save_pretrained(model_path)

运行结果

创建遵循指令的推理管道（Inference Pipeline)

可以看出，在第一次迭代中，用户提供的指令使用分词器转换为令牌ID，然后准备的输入提供给模型。该模型以 logits 格式为所有令牌生成概率 在预测概率上选择下一个令牌的方式由所选解码方法驱动。您可以在此博客中找到有关最流行的解码方法的更多信息。
有几个参数可以控制文本生成质量：
Temperature是用于控制 AI 生成文本中创造力水平的参数。通过调整Temperature，您可以影响 AI 模型的概率分布，使文本更加集中或多样化。考虑以下示例：AI 模型必须使用以下令牌概率完成句子 “The cat is ____.” ：
        playing: 0.5
        sleeping: 0.25
        eating: 0.15
        driving: 0.05
        flying: 0.05
oLow temperature (e.g., 0.2): AI模型变得更加集中和确定性，选择概率最高的令牌，例如 "playing."
oMedium temperature (e.g., 1.0): AI模型在创造力和专注力之间保持平衡，根据其概率选择令牌，没有明显的偏见，例如 "playing," "sleeping," or "eating."
oHigh temperature (e.g., 2.0): AI模型变得更加冒险，增加了选择不太可能的令牌的机会，例如 "driving" and "flying."

Top-p也称为核心采样 (nucleus sampling)，用于根据累积概率控制 AI 模型考虑的令牌范围的参数。通过调整Top-p 值，您可以影响 AI 模型的令牌选择，使其更加集中或多样化。对相同的示例 cat ，请考虑以下top_p设置：
oLow top_p (e.g., 0.5): AI 模型仅考虑累积概率最高的令牌，例如 "playing."
oMedium top_p (e.g., 0.8): AI 模型考虑具有较高累积概率的令牌，例如 "playing," "sleeping," and "eating."
oHigh top_p (e.g., 1.0): AI 模型考虑所有令牌，包括概率较低的令牌，例如 "driving" and "flying."

Top-k 是另一种流行的采样策略。与 Top-P 进行比较，Top-P 从累积概率超过概率 P 的最小可能单词组进行选择，在 Top-K 抽样中，K 最有可能被过滤掉下一个单词，并且概率质量仅在下一个 K 个单词之间重新分配。在我们的 cat 示例中，如果 k=3，则只有“playing”、“sleeping”和“eathing”将被考虑为下一个单词。

为了优化生成过程并更有效地使用内存，请启用 use_cache=True 选项。由于输出端是自动回归的，因此输出令牌隐藏状态在计算后每进一步生成步骤后保持不变。因此，每次想要生成新令牌时重新计算它似乎很浪费。使用缓存，模型会在计算后保存成隐藏状态。该模型仅在每个时长计算最近要生成的输出令牌的令牌，将保存的令牌重用于隐藏令牌。这将transformer模型的生成复杂度从 O(n^3) 降低到 O(n^2)。有关其工作原理的更多详细信息，请参阅本文。使用此选项，模型获取上一步的隐藏状态（缓存的attention keys和values）作为输入，此外还提供当前步骤的隐藏状态作为输出。这意味着对于所有后续迭代，只需提供从上一步获得的新令牌和缓存的键值即可获得下一个令牌预测。

生成周期重复，直到到达序列令牌的末尾，或者在生成最大令牌时中断。如前所述，我们可以启用打印当前生成的令牌，而无需等到整个生成完成使用Streaming API 时，它会将新令牌添加到输出队列，然后在它们准备就绪时打印(print)它们。

运行指令推理管道（run Pipeline)

输入设定

from threading import Thread
from time import perf_counter
from typing import List
import gradio as gr
from transformers import AutoTokenizer, TextIteratorStreamer
import numpy as np

为user prompt 准备模板

INSTRUCTION_KEY = "### Instruction:"
RESPONSE_KEY = "### Response:"
END_KEY = "### End"
INTRO_BLURB = (
    "Below is an instruction that describes a task. Write a response that appropriately completes the request."
)

# This is the prompt that is used for generating responses using *****ready trained model.  It ends with the response
# key, where the job of the model is to provide the completion that follows it (i.e. the response itself).
PROMPT_FOR_GENERATION_FORMAT = """{intro}

{instruction_key}
{instruction}

{response_key}
""".format(
    intro=INTRO_BLURB,
    instruction_key=INSTRUCTION_KEY,
    instruction="{instruction}",
    response_key=RESPONSE_KEY,
)

输出分析的帮助程序

def get_special_token_id(tokenizer: AutoTokenizer, key: str) -> int:
    """
    Gets the token ID for a given string that ha***een added to the tokenizer as a special token.

    When training, we configure the tokenizer so that the sequences like "### Instruction:" and "### End" are
    treated specially and converted to a single, new token.  This retrieves the token ID each of these key***ap to.

    Args:
        tokenizer (PreTrainedTokenizer): the tokenizer
        key (str): the key to convert to a single token

    Raises:
        RuntimeError: if more than one ID was generated

    Returns:
        int: the token ID for the given key
    """
    token_ids = tokenizer.encode(key)
    if len(token_ids) > 1:
        raise ValueError(f"Expected only a single token for '{key}' but found {token_ids}")
    return token_ids[0]

tokenizer_response_key = next((token for token in tokenizer.additional_special_tokens if token.startswith(RESPONSE_KEY)), None)

end_key_token_id = None
if tokenizer_response_key:
    try:
        end_key_token_id = get_special_token_id(tokenizer, END_KEY)
        # Ensure generation stops once it generates "### End"
    except ValueError:
        pass

主要生成功能代码

def run_generation(user_text:str, top_p:float, temperature:float, top_k:int, max_new_tokens:int, perf_text:str):
    """
    Text generation function
   
    Parameters:
      user_text (str): User-provided instruction for a generation.
      top_p (float):  Nucleus sampling. If set to < 1, only the **allest set of most probable tokens with probabilities that add up to top_p or higher are kept for a generation.
      temperature (float): The value used to module the logits distribution.
      top_k (int): The number of highest probability vocabulary tokens to keep for top-k-filtering.
      max_new_tokens (int): Maximum length of generated sequence.
      perf_text (str): Content of text field for printing performance results.
    Returns:
      model_output (str) - model-generated text
      perf_text (str) - updated perf text filed content
    """
   
    # Prepare input prompt according to model expected template
    prompt_text = PROMPT_FOR_GENERATION_FORMAT.format(instruction=user_text)
   
    # Tokenize the user text.
    model_inputs = tokenizer(prompt_text, return_tensors="pt")

    # Start generation on a separate thread, so that we don't block the UI. The text is pulled from the streamer
    # in the main thread. Adds timeout to the streamer to handle exceptions in the generation thread.
    streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
    generate_kwargs = dict(
        model_inputs,
        streamer=streamer,
        max_new_token***ax_new_tokens,
        do_sample=True,
        top_p=top_p,
        temperature=float(temperature),
        top_k=top_k,
        eos_token_id=end_key_token_id
    )
    t = Thread(target=ov_model.generate, kwargs=generate_kwargs)
    t.start()

    # Pull the generated text from the streamer, and update the model output.
    model_output = ""
    per_token_time = []
    num_tokens = 0
    start = perf_counter()
    for new_text in streamer:
        current_time = perf_counter() - start
        model_output += new_text
        perf_text, num_tokens = estimate_latency(current_time, perf_text, new_text, per_token_time, num_tokens)
        yield model_output, perf_text
        start = perf_counter()
    return model_output, perf_text

应用UI助手

def estimate_latency(current_time:float, current_perf_text:str, new_gen_text:str, per_token_time:List[float], num_tokens:int):
    """
    Helper function for performance estimation
   
    Parameters:
      current_time (float): This step time in seconds.
      current_perf_text (str): Current content of performance UI field.
      new_gen_text (str): New generated text.
      per_token_time (List[float]): history of performance from previous steps.
      num_tokens (int): Total number of generated tokens.
     
    Returns:
      update for performance text field
      update for a total number of tokens
    """
    num_current_toks = len(tokenizer.encode(new_gen_text))
    num_tokens += num_current_toks
    per_token_time.append(num_current_toks / current_time)
    if len(per_token_time) > 10 and len(per_token_time) % 4 == 0:
        current_bucket = per_token_time[:-10]
        return f"Average generaton speed: {np.mean(current_bucket):.2f} tokens/s. Total generated tokens: {num_tokens}", num_tokens
    return current_perf_text, num_tokens

def reset_textbox(instruction:str, response:str, perf:str):
    """
    Helper function for resetting content of all text fields
   
    Parameters:
      instruction (str): Content of user instruction field.
      response (str): Content of model response field.
      perf (str): Content of performance info filed
   
    Returns:
      empty string for each placeholder
    """
    return "", "", ""

def select_device(device_str:str, current_text:str = "", progress:gr.Progress = gr.Progress()):
    """
    Helper function for uploading model on the device.
   
    Parameters:
      device_str (str): Device name.
      current_text (str): Current content of user instruction field (used only for backup purposes, temporally replacing it on the progres***ar during model loading).
      progress (gr.Progress): gradio progress tracker
    Returns:
      current_text
    """
    if device_str != ov_model._device:
        ov_model.request = None
        ov_model._device = device_str
       
        for i in progress.tqdm(range(1), desc=f"Model loading on {device_str}"):
            ov_model.compile()
    return current_text

运行指令遵循管道(instruction-following pipeline)

from openvino.runtime import Core

core = Core()
available_devices = core.available_devices

examples = [
    "Give me recipe for pizza with pineapple",
    "Write me a tweet about new OpenVINO release",
    "Explain difference between CPU and GPU",
    "Give five ideas for great weekend with family",
    "Do Androids dream of Electric sheep?",
    "Who is Dolly?",
    "Please give me advice how to write resume?",
    "Name 3 advantages to be a cat",
    "Write instructions on how to become a good AI engineer",
    "Write a love letter to my best friend",
]

with gr.Blocks() as demo:
    gr.Markdown(
        "# Instruction following using Databricks Dolly 2.0 and OpenVINO.\n"
        "Provide insturction which describes a task below or select among predefined examples and model writes response that performs requested task."
    )

    with gr.Row():
        with gr.Column(scale=4):
            user_text = gr.Textbox(
                placeholder="Write an email about *****paca that likes flan",
                label="User instruction"
            )
            model_output = gr.Textbox(label="Model response", interactive=False)
            performance = gr.Textbox(label="Performance", lines=1, interactive=False)
            with gr.Column(scale=1):
                button_clear = gr.Button(value="Clear")
                button_submit = gr.Button(value="Submit")
            gr.Examples(examples, user_text)
        with gr.Column(scale=1):
            device = gr.Dropdown(choices=available_devices, value=current_device, label="Device")
            max_new_tokens = gr.Slider(
                minimum=1, maximum=1000, value=256, step=1, interactive=True, label="Max New Tokens",
            )
            top_p = gr.Slider(
                minimum=0.05, maximum=1.0, value=0.92, step=0.05, interactive=True, label="Top-p (nucleus sampling)",
            )
            top_k = gr.Slider(
                minimum=0, maximum=50, value=0, step=1, interactive=True, label="Top-k",
            )
            temperature = gr.Slider(
                minimum=0.1, maximum=5.0, value=0.8, step=0.1, interactive=True, label="Temperature",
            )

    user_text.submit(run_generation, [user_text, top_p, temperature, top_k, max_new_tokens, performance], [model_output, performance])
    button_submit.click(select_device, [device, user_text], [user_text])
    button_submit.click(run_generation, [user_text, top_p, temperature, top_k, max_new_tokens, performance], [model_output, performance])
    button_clear.click(reset_textbox, [user_text, model_output, performance], [user_text, model_output, performance])
    device.change(select_device, [device, user_text], [user_text])

if __name__ == "__main__":
    try:
        demo.launch(enable_queue=True, share=False, height=800)
    except Exception:
        demo.launch(enable_queue=True, share=True, height=800)

运行结果

范例1： Write instruction on how to become a good AI engineer

范例2： Who is Dolly?

范例2： Explain difference between CPU and GPU?

以下是右侧设备选择(GPU.0为iGPU;GPU.1为Intel Arc及各参数调控局部放大图)

结语：

Intel Arc显卡驱动安装